Method for operating a driver assistance device of a motor vehicle, driver assistance device and motor vehicle

ABSTRACT

The invention relates to a method for operating a driver assistance device ( 2 ) in a motor vehicle ( 1 ), in which an image ( 15 ) of an environmental region ( 12 ) of the motor vehicle ( 1 ) is captured by means of a camera ( 11 ) of the driver assistance device ( 2 ) and in addition sensor data is captured to the environmental region ( 12 ) by means of a sensor ( 3  to  6 ) different from the camera ( 11 ), wherein an object ( 16 ) located in the environmental region ( 12 ) is identified in the image ( 15 ) by means of an electronic computing device of the driver assistance device ( 2 ) and the sensor data of the sensor ( 3  to  6 ) is used for identifying the object ( 16 ) in the image ( 15 ).

The invention relates to a method for operating a driver assistancedevice in a motor vehicle. An image of an environmental region of themotor vehicle is captured by means of a camera of the driver assistancedevice. In addition, sensor data is captured to the environmental regionby means of a sensor different from the camera. In addition, theinvention relates to a driver assistance device for performing such amethod as well as to a motor vehicle with such a driver assistancedevice.

Driver assistance devices are already known from the prior art indiverse configuration. Thus, camera systems are known on the one hand,which have a plurality of video cameras attached to the motor vehicle,the images of which can be displayed on a display in the motor vehicle.The images of the cameras can also be subjected to image processing, andadditional functionalities can be provided based on the images. Forexample, based on the images, object identification is effected suchthat the camera system can serve as a collision warning system. On theother hand, systems are also known, which are formed for measuringdistances between the motor vehicle and the obstacles located in itsenvironment. Here, for example ultrasonic sensors are meant, which canbe disposed distributed on the front and the rear bumper of the motorvehicle. Each ultrasonic sensor then has its own capturing range, whichrepresents a partial segment of a common capturing range of the entireultrasonic sensor system. Thus, each ultrasonic sensor measures thedistances in its own capturing range.

It is also already prior art to combine a camera system with a sensorsystem in a motor vehicle. Such a sensor fusion is for example knownfrom the document GB 2463544 A. Here, a plurality of ultrasonic sensorsis employed, which are for example attached to a bumper. Theenvironmental region of the motor vehicle detected by the ultrasonicsensors is additionally imaged by means of a camera. A computing deviceprocesses the sensor data of the ultrasonic sensors as well as theimages of the camera at the same time. On the one hand, the images aredisplayed on a display in the motor vehicle; on the other hand, it isexamined by means of the computing device if a detected objectapproaches the motor vehicle. As the case may be, a warning signal isthen output.

In the prior art, thus, the sensor fusion is effected such that allinformation of different sensor systems—namely of the camera on the onehand and the ultrasonic sensors on the other hand—is collected andprocessed at the same time in a common computing device.

It is the object of the invention to demonstrate a solution how in amethod of the initially mentioned kind the images of the camera on theone hand and the sensor data of the sensor on the other hand can bebetter combined with each other than in the prior art.

According to the invention, this object is solved by a method, by adriver assistance device as well as by a motor vehicle having thefeatures according to the respective independent claims. Advantageousimplementations of the invention are the subject matter of the dependentclaims, of the description and of the figures.

A method according to the invention serves for operating a driverassistance device of a motor vehicle by capturing an image of anenvironmental region of the motor vehicle by means of a camera of thedriver assistance device as well as by capturing sensor data to theenvironmental region by means of a sensor different from the camera,namely for example an ultrasonic sensor. According to the invention, itis provided that an object located in the environmental region isidentified in the image by means of an electronic computing device ofthe driver assistance device and the sensor data of the sensor is usedfor identifying the object in the image.

Thus, the effect according to the invention is achieved in that thecomputing device identifies the object external to vehicle in the imagenot or not only based on the image data, but (also) based on the sensordata of the at least one sensor. Therein, the invention is based on therealization that with the aid of the detection algorithms known from theprior art, which serve for detecting objects based on images, it is notalways possible to identify the object in the captured image. Namely,object identification solely based on the image data is not possible oronly in restricted manner in particular in a near range of up to about0.5 m from the motor vehicle. It can occur that an object located inthis near range is depicted in the captured images, but cannot beidentified solely based on the images. Now, the invention takes the wayto use the sensor data of the sensor for identifying the object in thecaptured image. For example, this can be configured such that, if theobject cannot be identified based on the image data, the same object isidentified solely based on the sensor data. By contrast, if the objectis identified both based on the sensor data and based on the image,thus, the identification of the object in the image can be effected bothdepending on the sensor data and depending on the image data. Overall,thus, the sensor fusion is improved compared to the prior art, and theaccuracy and reliability of the object identification in the image ofthe camera are increased.

The identification of the object in the image can for example beeffected such that at least one region of the object depicted in theimage is surrounded by a bounding box. Such an approach to label anobject identified based on image data in the image by means of abounding box is already known for example from the printed matter JP2011/119917 A. In this embodiment, however, it is also proposed togenerate such a bounding box not or not only based on the image data ofthe camera, but additionally or alternatively also based on the sensordata of the sensor. This embodiment exploits the fact that a sensoroperating according to the echo propagation time method has a certaindetection range and measures the distances only in this detection range.In particular with ultrasonic sensors, this detection range isrelatively narrow such that with the presence of a plurality of sensorswith good accuracy, the position of the object relative to the motorvehicle and therefore also the position of the object in the capturedimage can be determined, too. The bounding box generated based on thesensor data can for example have a width in the image of the camera,which corresponds to the width of the detection range of the sensor.Such a camera image with the bounding box can then be used in verydifferent manners: on the one hand, this image can be displayed on adisplay in the motor vehicle such that the driver is informed about thedetected object. On the other hand, this image with the bounding box canalso be transmitted to other driver assistance systems in the motorvehicle and these other systems can use the image for providingdifferent functionalities in the motor vehicle. Such a system can forexample be the collision warning system, which is able to generate awarning signal for warning the driver based on the image.

In an embodiment it is provided that the object is identified in theimage both based on the sensor data of the sensor and based on the imageof the camera by means of a computing device. This is in particularprovided if the object identification is possible both based on thesensor data and based on the image data of the camera, thus if theobject is located in an overlapping region between the detection rangeof the sensor as well as an image analysis range, in which the objectidentification is also possible based on the image data. This embodimenthas the advantage that the object external to vehicle can be identifiedparticularly reliably and extremely exactly in the image of the camera.Namely, this embodiment combines the advantages of the objectidentification based on the image data on the one hand with theadvantages of the object identification based on the sensor data on theother hand such that the respective disadvantages of the two objectidentification methods can be avoided.

For example, this can be effected such that in the image a firstbounding box is generated based on the sensor data of the sensor, whilea second bounding box is generated based on the image of the camera(thus by means of image processing). Then, the two bounding boxes can bemerged to a common bounding box. Thus, the generation of the boundingbox in the image of the camera is particularly precise.

Particularly preferred, the identification of the object involves that awidth of the object in the image is determined based on the image of thecamera, while the position of a lower end of the object in the image isdetermined based on the sensor data of the sensor. This embodiment isbased on the realization that both the object identification based onthe image data and the identification based on the sensor data have“weak points”. Thus, in the object identification based on the imagedata, the exact determination of the lower end in the image is notpossible or only in restricted manner due to the used detectionalgorithms (optical flow, ego-motion compensation method). With thesedetection algorithms, for example, the feet of pedestrians can onlyinexactly be detected. On the other hand, the determination of the widthof the object in the image is only possible with restricted accuracybased on the sensor data of the sensor. For this reason, presently it isproposed to use the image data of the camera for determining the widthof the object in the image and to use the sensor data of the sensor fordetermining the position of a lower end of the object in the image. Therespective disadvantages of the identification methods—based on theimage data on the one hand and based on the sensor data on the otherhand—can therefore be avoided, and the object identification can beparticularly precisely effected.

The latter embodiment can for example be realized such that the mergingof the two bounding boxes in performed in a very specific manner: forthe common bounding box, the width of the second bounding box (based onthe camera data) as well as the position of a lower edge of the firstbounding box in the image (based on the sensor data) can be adopted. Thecommon bounding box thus has the width of the bounding box generatedbased on the image data, and the position of the lower edge of thecommon bounding box corresponds to the position of the bounding boxgenerated based on the sensor data. The common bounding box thusparticularly precisely reflects the actual position of the object in theimage.

As already explained, it can occur that the object is within thedetection range of the sensor, but outside of an image analysis range,in which the identification of the object based on the image data ispossible at all. In such a case, the same object is preferablyidentified in the image solely based on the sensor data of the sensor.The deficiencies of the object identification based on the sensor datais accepted in this embodiment. However, this embodiment allows thateven in absence of object identification based on the image data, theobject external to vehicle nevertheless can be identified in the imageof the camera.

By contrast, if the object is outside of the detection range of thesensor, the same object is identified in the image solely based on theimage of the camera. Thus, if the object cannot be identified based onthe sensor data, solely the image data of the camera is used for objectidentification. This is in particular the case if the object isrelatively far from the motor vehicle, namely in a distance of greaterthan for example 2.2 m. Namely, in such a distance, the object can nolonger be detected with the aid of the sensor, and the objectidentification can be performed solely based on the image data.

An ultrasonic sensor is preferred, which is used for capturing thesensor data to the environmental region of the motor vehicle. Overall, aplurality of ultrasonic sensors can be used, which can be disposeddistributed on the front bumper and/or on the rear bumper of the motorvehicle. Each ultrasonic sensor then has its own detection range, andthe individual detection ranges can be next to each other—optionallyalso overlapping. However, the invention is not restricted to anultrasonic sensor. Other sensors can also be employed, which aredifferent from the camera. In particular, the at least one sensor issuch one, which operates according to the echo propagation time method,thus a distance sensor, in which the distances are measured by measuringthe propagation time of the transmit signal.

The invention also relates to a driver assistance device for a motorvehicle, which is formed for performing a method according to theinvention.

A motor vehicle according to the invention has a driver assistancedevice according to the invention.

The preferred embodiments presented with respect to the method accordingto the invention and the advantages thereof correspondingly apply to thedriver assistance device according to the invention as well as to themotor vehicle according to the invention.

Further features of the invention are apparent from the claims, thefigures and the description of figures. All of the features and featurecombinations mentioned above in the description as well as the featuresand feature combinations mentioned below in the description of figuresand/or shown in the figures alone are usable not only in therespectively specified combination, but also in other combinations orelse alone.

Now, the invention is explained in more detail based on a preferredembodiment as well as with reference to the attached drawings.

There show:

FIG. 1 in schematic illustration a plan view of a motor vehicleaccording to an embodiment of the invention;

FIG. 2 in schematic illustration an image of a camera, wherein detectionranges of sensors are illustrated in the image;

FIG. 3 in schematic illustration the image according to FIG. 2, whereinthe generation of a bounding box based on sensor data is explained inmore detail;

FIG. 4 the image with the bounding box, which is generated solely basedon the sensor data;

FIG. 5 the image with a bounding box, which is generated solely based onthe image (thus the image data);

FIG. 6 the image with the two bounding boxes;

FIG. 7 in enlarged illustration the two bounding boxes;

FIG. 8 a section of the image with a common bounding box, which isgenerated from the two mentioned bounding boxes; and

FIG. 9 in schematic illustration the motor vehicle, wherein a methodaccording to an embodiment of the invention is explained in more detail.

A motor vehicle 1 illustrated in FIG. 1 according to an embodiment ofthe invention is a passenger car. The motor vehicle 1 includes a driverassistance device 2, which serves for assisting the driver in drivingthe motor vehicle 1. The driver assistance device 2 has a plurality ofultrasonic sensors 3, 4, 5, 6, which are disposed distributed on a rearbumper of the motor vehicle 1 in the embodiment. In the embodiment, fourultrasonic sensors 3 to 6 are provided in total. The number as well asthe arrangement of the ultrasonic sensors 3 to 6 is only exemplarilyillustrated in the example according to FIG. 1; the number and thearrangement of the ultrasonic sensors 3 to 6 can be different accordingto embodiment. Thus, the ultrasonic sensors 3 to 6 can also be disposedon a front bumper of the motor vehicle 1. The configuration of thesensors 3 to 6 as ultrasonic sensors is also exemplary here. Othersensors can also be employed, which can measure distances between themotor vehicle 1 and the obstacles located in its environment. Forexample, radar devices or else optical distance sensors can also beused.

Each ultrasonic sensor 3 to 6 has a detection range 7, 8, 9, 10, inwhich the respective ultrasonic sensor 3 to 6 can measure the distances.If for example an object external to vehicle is in the detection range 7of the ultrasonic sensor 3, thus, the ultrasonic sensor 3 can detect thedistance of this object from the motor vehicle 1. The detection ranges 7to 10 are closely next to each other and immediately adjoin each other.The detection ranges 7 to 10 thus cover a relatively large environmentalregion behind the motor vehicle 1 such that the individual detectionranges 7 to 10 each represent a partial segment of the environmentalregion behind the motor vehicle 1. Therein, the respective detectionranges 7 to 10 are relatively narrow segments, which are next to eachother in vehicle transverse direction and are configured elongated invehicle longitudinal direction.

In addition, the driver assistance device 2 has a camera 11, which isdisposed in the rear region of the motor vehicle 1 similar to theultrasonic sensors 3 to 6 and images an environmental region 12 behindthe motor vehicle 1. The environmental region 12 imaged with the camera11 also includes the detection ranges 7 to 10 of the ultrasonic sensors3 to 6 such that the detection ranges 7 to 10 are within the imagedenvironmental region 12.

The camera 11 is a video camera, which is able to provide a plurality offrames per second or a temporal sequence of images. The camera 11 has arelatively large capturing angle or aperture angle, which even can be ina range of values from 120° to 190°. This angle is bounded by two lines13, 14 in FIG. 1, namely such that the imaged environmental region 12behind the motor vehicle 1 is also bounded by the bounding lines 13, 14.The camera 11 can for example be the so-called fish eye camera.

Both the ultrasonic sensors 3 to 6 and the camera 11 are electricallyconnected to an electronic computing device not illustrated in moredetail in the figures, which can for example include a digital signalprocessor and a memory. Thus, the computing device receives the sensordata of the ultrasonic sensors 3 to 6 on the one hand and also theimages—thus the image data—of the camera 11 on the other hand.

An exemplary image 15 of the camera 11, in which the environmentalregion 12 is depicted, is illustrated in FIG. 2. Because the coordinatesof the ultrasonic sensors 3 to 6 and of the camera 11 in the coordinatesystem of the motor vehicle 1 and in the coordinate system of the camera11, respectively, are known in the computing device, the positions ofthe detection ranges 7 to 10 in the image 15 are also known, as isschematically illustrated in FIG. 2. If an object 16 (here a pedestrian)exists within the detection range 7 to 10 of one of the ultrasonicsensors 3 to 6 (exemplarily in the detection range 8 of the ultrasonicsensor 4 in FIG. 2), a region 17 can be defined in the image 15, inwhich the imaged object 16 is located in the image 15. This is possiblebecause the position of the detection ranges 7 to 10 in the image 15 ofthe camera 11 is known in the computing device. If the object 16 isdetected by means of the ultrasonic sensor 4 (if this ultrasonic sensor4 provides a measured distance), it can be assumed as sure that theobject 16 is actually located in the detection range 8 of the ultrasonicsensor 4. The defined region 17, in which the object 16 is located, canhave the same width in the image 15 as the detection range 8 of theultrasonic sensor 4. Because the distances to the object 16 can beparticularly precisely measured by means of the ultrasonic sensor 3 to6, the position of a lower end 18 of the object 16 in the image 15 canbe determined based on the measured distance. In the embodimentaccording to FIG. 3, the lower end 18 corresponds to a foot of thepedestrian. On the other hand, it is not readily possible to determinethe width of the object 16 in the image 15 solely based on the sensordata of the ultrasonic sensors 3 to 6. Therefore, the region 17 isdefined with such a width, which corresponds to the width of thedetection range 8.

Solely based on the sensor data of the ultrasonic sensors 3 to 6, thus,a first bounding box 19 can be generated in the image 15, which isillustrated in FIG. 4 and corresponds to the region 17 according to FIG.3. The imagined detection ranges 7 to 10 of the ultrasonic sensors 3 to6 are not illustrated in FIG. 4 anymore. Thus, an image 15 according toFIG. 4 is available, in which the object 16 is surrounded by means ofthe bounding box 19.

Therefore, the object identification is here effected solely based onthe sensor data of the ultrasonic sensors 3 to 6. In order to generatethe first bounding box 19, namely, a special image processing of theimage 15 is not required.

This type of object identification, which is performed solely based onthe sensor data of the ultrasonic sensors 3 to 6, is for exampleprovided if the computing device is not capable of identifying theobject 16 in the image 15 solely based on the image processing of theimage data due to the low distance of the object 16. If the opticalobject identification does not provide results, thus, the object 16 inthe image 15 is detected—as shown above—solely based on the sensor dataof the ultrasonic sensors 3 to 6. Such an image 15 according to FIG. 4can then for example be displayed on a display in the motor vehicle 1 orelse be forwarded to further assistance systems.

By contrast, if the detection of the object 16 based on the camera datais possible, thus, the image processing algorithms known from the priorart can also be used, which serve for detecting the object in 16 in theimage 15. Such detection algorithms also provide a bounding box 20(second bounding box) as it is illustrated in more detail in FIG. 5. Theimage 15 according to FIG. 5 is therefore the result of a detectionalgorithm, by which the object 16 is identified. As is apparent fromFIG. 5, the actual width of the object 16 is better reflected with thesecond bounding box 20 than with the first bounding box 19 (see FIG. 4).However, a disadvantage in the object identification based on the imagedata is in that the base of the object 16—thus the lower end 18 (herethe feet)—cannot be uniquely identified by the computing device. Namelyfor the reason that usually the so-called “optical flow” is used as thedetection algorithm, which includes the ego-motion compensation. Thisalgorithm outputs the image 15 with the bounding box 20, wherein thelower edge of the bounding box 20 in the image 15 does not alwayscorrespond to the position of the lower end 18 in the image 15.

If the object identification based on the sensor data of the ultrasonicsensors 3 to 6 is not possible, thus, the detection of the object 16 iseffected solely based on the image 15, thus solely based on the cameradata. The result of this object identification is illustrated in FIG. 5.

It can also occur that the object 16 in the image 15 can be identifiedboth based on the sensor data of the ultrasonic sensors 3 to 6 and basedon the image data of the camera 11. As is illustrated in FIG. 6, thus,two bounding boxes 19, 20 are generated, namely the first bounding box19 based on the sensor data and the second bounding box 20 based on theimage processing. From the two bounding boxes 19, 20, now, a commonbounding box 21 can be generated (see FIG. 8). The generation of thecommon bounding box 21 is effected such that the width of the secondbounding box 20 on the one hand and the position of the lower edge ofthe first bounding box 19 on the other hand are adopted. With referenceto FIG. 7, in the computing device, the second bounding box 20 is takenand expanded in its height such that a lower edge 22 of the secondbounding box 20 coincides with a lower edge 23 of the first bounding box19. Thus, the height of the second bounding box 20 is increased bydisplacing the lower edge 22 in the image 15 downwards until this edge22 reaches the lower edge 23. Therein, the upper edge of the boundingbox 20 remains unchanged. Such an altered bounding box 20 thencorresponds to the common bounding box 21 according to FIG. 8, which wasgenerated based on the two bounding boxes 19, 20. Thus, the bounding box21 according to FIG. 8 represents the result of the objectidentification both based on the image data and based on the sensordata.

As already explained, different situations can occur:

-   -   The object identification can be possible exclusively based on        the sensor data;    -   the object identification can be possible exclusively based on        the image data; and    -   both types of object identification can be possible (common        bounding box 21).

Usually, this will depend on in which distance the object 16 is locatedfrom the motor vehicle 1. With reference to FIG. 9, three differentregions 24, 25, 26 can be defined in total, which are each in differentdistances from the vehicle tail. Therein, the first region 24 extendsfor example up to a distance of 0.5 m from the motor vehicle 1. Thesecond region 25 for example extends from 0.5 m to 2.2 m from the motorvehicle 1. The third region 26 extends from 2.2 m from the motor vehicle1. In the first region 24, basically, object identification is onlypossible based on the sensor data. By contrast, in the second region 25,the object identification is possible both based on the image data andbased on the sensor data. Finally, in the third region 26,identification is exclusively possible based on the camera data. If theobject 16 is in the first region 24, thus, the first bounding box 19 canbe generated within the scope of the identification of the object 16.However, if the object 16 is in the region 25, thus, the common boundingbox 21 can be generated from the bounding boxes 19, 20. Finally, if theobject 16 is in the third region 26, thus, only the second bounding box20 is generated.

Thus, at the end, an image 15 with a bounding box 19, 20 or 21 isavailable. This image 15 can now be displayed on a display. Additionallyor alternatively, this image 15 can also be further processed in orderto be able to provide further functionalities in the motor vehicle 1,namely for example the functionality of warning the driver.

1. Method for operating a driver assistance device (2) in a motorvehicle (1) by capturing an image (15) of an environmental region (12)of the motor vehicle (1) by means of a camera (11) of the driverassistance device (2), and capturing sensor data to the environmentalregion (12) by means of ultrasonic sensors (3 to 6) with an individualdetection range (7 to 10) covering partial segment of the environmentalregion (12), characterized in that an object (16) located in theenvironmental region (12) is identified in the image (15) by means of anelectronic computing device of the driver assistance device (2), and thesensor data of the sensors (3 to 6) is used for identifying the object(16) in the image (15).
 2. Method according to claim 1, characterized inthat the identification of the object (16) in the image (15) includesthat at least one region of the object (16) depicted in the image (15)is surrounded by a bounding box (19, 20, 21).
 3. Method according toclaim 1 or 2, characterized in that the object (16) is identified in theimage (15) both based on the sensor data of the sensors (3 to 6) andbased on the image (15) of the camera (11) by means of the computingdevice.
 4. Method according to claims 2 and 3, characterized in that inthe image (15) a first bounding box (19) is generated based on thesensor data of the sensors (3 to 6) as well as a second bounding box(20) is generated based on the image (15) of the camera (11) and the twobounding boxes (19, 20) are combined to a common bounding box (21). 5.Method according to claim 3 or 4, characterized in that theidentification of the object (16) includes that a width of the object(16) is determined based on the image (15) of the camera (11) and theposition of a lower end (18) of the object (16) is determined in theimage (15) based on the sensor data of the sensors (3 to 6).
 6. Methodaccording to claims 4 and 5, characterized in that the combination ofthe two bounding boxes (19, 20) includes that the width of the secondbounding box (20) and the position of a lower edge (23) of the firstbounding box (19) in the image (15) are used for the common bounding box(21).
 7. Method according to anyone of the preceding claims,characterized in that if the object (16) is within a detection range (7to 10) of the sensor (3 to 6), to which the sensor data is captured, andoutside of an image analysis range (25, 26), in which the identificationof the object (16) based on the image (15) is possible, the same object(16) is identified in the image (15) solely based on the sensor data ofthe sensors (3 to 6).
 8. Method according to anyone of the precedingclaims, characterized in that if the object (16) is outside of adetection range (7 to 10) of the sensors (3 to 6), the same object (16)is identified in the image (15) solely based on the image (15) of thecamera (11).
 9. Method according to anyone of the preceding claims,characterized in that an ultrasonic sensor is used as the sensors (3 to6).
 10. Driver assistance device (2) for a motor vehicle (1), includingat least one camera (11) for capturing images (15) of an environmentalregion (12) of the motor vehicle (1), including ultrasonic sensors (3 to6) with an individual detection range (7 to 10) covering partial segmentof the environmental region (12) for capturing sensor data to theenvironmental region (12), and including an electronic computing devicefor processing the images (15) and the sensor data, characterized inthat the computing device is arranged to perform a method according toanyone of the preceding claims.
 11. Motor vehicle (1) including a driverassistance device (2) according to claim 10.